Recommendations for a standardised educational program in robot assisted gynaecological surgery: Consensus from the Society of European Robotic Gynaecological Surgery (SERGS).

Background The Society of European Robotic Gynaecological Surgery (SERGS) aims at developing a European consensus on core components of a curriculum for training and assessment in robot assisted gynaecological surgery. Methods A Delphi process was initiated among a panel of 12 experts in robot assisted surgery invited through the SERGS. An online questionnaire survey was based on a literature search for standards in education in gynaecological robot assisted surgery. The survey was performed in three consecutive rounds to reach optimal consensus. The results of this survey were discussed by the panel and led to consensus recommendations on 39 issues, adhering to general principles of medical education. Results On review there appeared to be no accredited training programs in Europe, and few in the USA. Recommendations for requirements of training centres, educational tools and assessment of proficiency varied widely. Stepwise and structured training together with validated assessment based on competencies rather than on volume emerged as prerequisites for adequate and safe learning. An appropriate educational environment and tools for training were defined. Although certification should be competence based, the panel recommended additional volume based criteria for both accreditation of training centres and certification of individual surgeons. Conclusions Consensus was reached on minimum criteria for training in robot assisted gynaecological surgery. To transfer results into clinical practice, experts recommended a curriculum and guidelines that have now been endorsed by SERGS to be used to establish training programmes for robot assisted surgery.

Recommendations for a standardised educational program in robot assisted gynaecological surgery: Consensus from the Society of European Robotic Gynaecological Surgery (SERGS)

Introduction
The introduction of robotics in gynaecological surgery has resulted in a need for new surgical skills and a requirement for a syllabus, training structure and assessment model. Professional societies should, in parallel to training programs in conventional endoscopy (Campo et al., 2016), also develop programs for safe and efficient training in specific areas such as robotic surgery.
Laparoscopic surgery was introduced in the late sixties although it was not until recently that regulatory authorities realised that the traditional master-apprentice principle was insufficient to provide safe and adequate skills and to monitor proficiency (Stassen et al., 2010). Consequently, there has been criticism on the way surgeons are trained (Li et al., 2016;Sridhar et al., 2017;Beane et al., 2019). Furthermore, deficient training and credentialing predisposes to litigation (Lee et al., 2011). Trainees perceive that training in laparoscopic surgery, and in particular robotics, is poor (Gan et al., 2017). This has led to a call for more structured and validated training and more virtual instruction (Schreuder et al., 2012;Beane, 2019).
Although curricula for training in conventional laparoscopy are developing (Tremblay et al., 2014), this is evolving more slowly for robot-assisted surgery (Fisher et al., 2015). Nevertheless, the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES), together with the Minimal Invasive Robotic Association (MIRA) drafted a position paper in 2007 (Herron et al., 2008). This resulted in the first curriculum, the Fundamentals of Robotic Surgery (FRS) in the USA (Smith et al., 2014). The European Board and College of Obstetricians and Gynaecologists (EBCOG) issued 'Robotic Surgery Standards' as part of their 'Gynaecology Standards' (Mahmood et al., 2014). This latter document describes training in broad terms only, but defines the learning curve of surgeons that should be 'specifically trained' for robot-assisted procedures.
Urologists were the first in Europe to propose a curriculum. The syllabus developed by the European Association of Urologists (EAU) Robotic Urology Section (ERUS) is the only curriculum that encompasses the complete pathway from technical instruction to patient procedures (Herron and Marohn, 2008). The Society of European Robotic Gynaecological Surgery (SERGS) also aimed to develop guidelines for the safe introduction of robotassisted surgery although consensus was lacking on many issues (Rusch et al., 2018). A Delphi process which is described in this paper was necessary to finalise a curriculum that is proposed to be used for robotic gynaecological surgery.

Material and methods
In January 2017 an expert advisory committee was formed to formulate a consensus on recommendations for education in robotic gynaecological surgery. Fifteen experienced surgeons and members of SERGS were invited and eleven accepted. A fellow trainee (PR) was invited also, along with a member of the EAU (JWC) who had experience in guiding such process.
The strategy was divided into two parts. The first was a systematic literature review (Figure 1). A search was undertaken using Pubmed and Medline with the key terms "robotic", "training", "gynecology", "surgery" AND "assisted". Articles selected included single-centre series, meta-analyses, randomised controlled trials (RCTs) and systematic reviews between 2007 and 2017. The search yielded a total of 104 potential studies, of which 51 focused on training, testing or credentialing in robotic assisted gynaecological surgery. These papers were then screened for key questions divided in subgroups on four main subjects, (a) qualification/ credentialing, (b) course/content of robotic training, (c) methodology/structure of robotic training and (d) testing/test instruments.
In the second phase, the literature review lead to formulating key questions for a Delphi survey (Thangaratinam, 2005). The aim was to achieve agreement on 39 aspects of a robotic curriculum

No. Question/Answer
Consensus 0. Curriculum -General Agreement 1 Q: Do you agree that a standardised robotic training curriculum for gynaecology will be advantageous to robotic training?
A standardised robotic training curriculum for gynecology will be advantageous to robotic training (100%).

Qualification
Trainer & Trainee 2 Q: Experienced surgeons are exempt from completing the advanced procedural training assessment. But should learn about the basic training in new robotic systems, if they are using a new system?
Experienced surgeons are exempt from completing the advanced procedural training assessment; but they should learn about the basic training in new robotic systems, if they are using a new system (100%).
3 Q: Experienced surgeons should still study and be tested on the advanced robotic curriculum? 50%; failed 4 Q: What is the minimum number of cases that a trainee should be mentored/proctored by an experienced trainer before they are independent surgeons?
The minimum number of cases that a trainee should be mentored/proctored by an experienced trainer before they are independent surgeons is 10 cases (80%).
13 Q: Baseline evaluation should include which parts/stages (can tick multiple answers as required).
23 Q: Do you agree that there should be a common approach for basic robotic skills training with a similar pathway across subspecialty groups?
Agreement that there should be a common approach for basic robotic skills training with a similar pathway across subspecialty groups (90%).

Course/Sequence
24 Q: Is a stepwise approach (modular training) to an index procedure advantageous to training?
A stepwise approach (modular training) to an index procedure is regarded advantageous (100%).
25 Q: Is an index procedure, which should be mastered within a given period of time, necessary?
An index procedure mastered within a given period of time is necessary (80%).
26 Q: If so, do you agree that for benign gynecology a suitable index procedure would be?
A suitable index procedure for benign gynecology would be benign hysterectomy (90%).
27 Q: If so, do you agree that for gynecology oncology a suitable index procedure would be?
A suitable index procedure for gynecological oncology would be pelvic lymphadenectomy (80%).
28 Q: Is a resident experienced trainer/proctor necessary when the trainee is proceeding to 'transition to full procedure' in the surgeons home institution?
A resident experienced trainer/proctor is necessary when the trainee is proceeding to "transition to full procedure" in the surgeons home institution (100%).

E-Learning
Q: Each section of the e-learning should have questions to evaluate knowledge.
Each section of the e-learning should have questions to evaluate knowledge (90%).

Q: Advanced e-learning modules should be evaluated with online tests?
Advanced E-learning modules should be evaluated with online tests (100%).
Evaluation, Analysis 31 Q: Non-technical skills training should be evaluated with a scoring system?
Non-technical skills training should be evaluated with a scoring system (80%).
32 Q: Non-technical skills can be sufficiently assessed with NOTSS (Non-Technical Skills for Surgeons)?
33 Q: Would trainees benefit from validated scoring systems to provide more consistent feedback?
Trainees would benefit from validated scoring systems to provide more consistent feedback (90%).
34 Q: Should full procedure technique be evaluated with a submitted video to certified independent examiners?
Full procedure technique should be evaluated with a submitted video to certified independent examiners (80%).
35 Q: If answer to above yes, which case number should be sent for analysis and feedback?
70%, failed 36 Q: Evaluation of videos should be completed with a validated standardised scoring system?
Evaluation of videos should be completed with a validated standardized scoring system (80%).  (Siddique et al., 2016). The recognition of 'centres of excellence' might assist although criteria for such centres are not defined. In general, it is assumed that high volume units qualify as training centres (Gastrich et al., 2011;Gobern et al., 2013).

Content of training, including courses
Forty-five of the 51 papers addressed this issue. Structured and standardised training with pre-set learning goals is paramount to accomplish training in a timely and thorough fashion . Modules (see Qualification/Credentialing) of training lead from e-learning, to virtual training, to model training, and finally to procedural training ( Figure 2, modified for gynaecology after Volpe et al. (2015)). E-learning tools are considered as a basis for basic and advanced training (Maertens et al., 2016). In later stages of practice, e-learning may provide a resource for permanent training by sharing information provided by professionals themselves (www.websurg.com; https://eacademy.esgo.org/). Virtual training may teach technical skills in a simulated and safe environment and provide tools for objective assessment. Validated systems are commercially available (Abboudi et al., 2013;Moglia et al., 2016). emerging from this review and from the experts' experience (Table I). An internet survey was generated and sent to panel members. The questionnaire was completed over three rounds. Google Forms® analytical software was used to record and measure consensus levels of the e-consensus at each round. Results were displayed as percentages so they could be reflected on before selecting a response in subsequent rounds. In the second and third rounds questions on which an 80% consensus was reached, were removed. Cronbach alpha was chosen as a measure of consistency. A cutoff value of 0.8 was chosen to determine consensus. After the three rounds a meeting was held to present results focusing on those questions that had not reached a 0.8-consensus. The final manuscript was reviewed and approved by the SERGS Council.

Results of Evidence Synthesis
The literature review resulted in 104 papers of which 51 addressed the need for training or a curriculum with attention to 1) qualification/credentialing, 2) content of training, 3) methodology/structure of training and 4) testing/test instruments (Table II).

Qualification/Credentialing
Of 51 papers selected, 25 contained credentialing recommendations. Some authors advised against definitions for a centre to be accredited . Others underlined that by avoiding such clarity, centres cannot evade their responsibility for correct introduction and safe use of robotic surgery systems (Pradarelli et al., 2017). These authors argued that training standards promoted by manufacturers were insufficient. In contrast, surgeons and hospitals were obliged to develop educational strategies to keep up with new surgical advances while considering their duty of care to patients.
Institutions are responsible for governance, including repetitive re-assessments to maintain surgical privileges (Committee opinion no. 628,2015). A recent Canadian study showed considerable 37 Q: Scoring systems for video analysis should include (can tick multiple boxes)?
Scoring systems for video analysis should include a combination of subjective and objective scoring systems (e.g. GEARS, OSATS, a new objective scoring system) (100%).
38 Q: How many 'experts' should analyse the surgery videos? 2 experts should analyse the surgery videos (90%).
39 Q: Should video analysis and the logbook be the final evaluation step for 'certification'?
Video analysis and the logbook should be the final evaluation step for certification (90%).    Construct validation (whether the exercise is discriminatory) and face validation (whether the exercise resembles real-life) need to have been assessed (Schreuder et al., 2014). Model training may teach technical skills in a more realistic environment by working in a box trainer, an animal model, or a cadaver (Sridhar et al., 2017). An important and final part is procedural training in virtual and in-vivo procedures. Clinical procedures should be performed under the guidance of expert tutors (Sluis van der et al., 2013;Sandadi et al., 2014;Sridhar et al., 2017). In the ERUS experience, a modular sequential introduction to complex procedures was the safest and most effective way to learn complex surgery. Rather than starting and finishing a whole procedure at once, modular training takes the trainee stepwise through well-defined structured stages. This approach ensures maximum attention for each step, avoiding concentration loss during long procedures with multiple parts (Crane et al., 2013;Letouzey et al., 2014;Lovegrove et al., 2016;Carter-Brooks et al., 2018).

Methodology/Structure of training
Of the 51 papers, 33 included guidance for structured tuition, either in modular training (n=11), stepwise learning (n=7), or both (n=15). Training in complex procedures using sophisticated technology requires systematic, structured and (therefore) modular training (Schreuder and Verheijen, 2009;Ng et al., 2011;Letouzey et al., 2014). This has been developed and validated by ERUS for prostatectomy (Volpe et al., 2015). The literature search showed a plea for curricula to be built up from e-learning, through virtual and box training to artificial and animal model teaching (see also Content of training, including courses) (Volpe et al., 2015).

Testing/Test Instruments
Not all papers that discussed the content and structure of training defined relevant and measurable end points. Only 29 of the 51 papers gave recommendations for assessment of training.

Competency based assessment
Competence based training with structural assessment has been introduced in the curriculum for general gynaecology successfully (Boerebach et al., 2016) . The Royal College of Physicians and Surgeons of Canada described seven competencies of a physician which included 'professional', 'communicator', 'collaborator', 'leader', 'health advocate', 'scholar', and 'medical expert' as central roles (Frank et al., 2015). Evaluation of these roles is now integrated in the assessment of general obstetrics and gynaecology training in the United Kingdom and Netherlands (Garofalo and Aggarwal, 2017). In a technical field such as robotics, these competencies are essential for a future expert and team to evolve (Sananès et al.,2011;Payne and Pitter, 2011;Schreuder et al., 2014).

Structured assessment
A regular, non-judgemental and objective evaluation of progress is regarded essential for effective learning and patient safety. Pre-and post-testing at various modules help to develop skills (Thomaier et al., 2017).
Systematic assessment after each module or parts thereof should monitor progression. Structured assessment enables the tutor to systematically review skills and competencies. Objective and quantitative scoring can be performed using the Global Evaluative Assessment of Robotic Skills (GEARS) (Goh et al., 2012) and Objective Structured Assessment of Technical Skills (OSATS) (Faulkner et al., 1996). GEARS is the only instrument designed and validated for robot-assisted surgery (Sánchez et al., 1996;Goh et al., 2012).
To integrate non-technical competencies Nontechnical Skills for Surgeons (NOTSS) has been developed (Flin et al., 2006). At the end of training, assessment of an unedited video of a procedure performed by the trainee should be part of a final evaluation (Payne and Pitter, 2011;Hoffman et al., 2012a,b;Vaccaro et al., 2013). This allows appraisal by an independent assessor using tools like GEARS (Tunitsky et al., 2013). Video assessment is now need to be assessed in advanced procedural training if they were familiar with their platform (Q2). However, there was a vote for trainers to have basic training if they changed platforms (Q2).
Whilst there was consensus that surgeons continue to report their outcomes after certification, there was no consensus on a reporting template to be used (Q6). In general, it was recommended to include patient specific details, comorbidities, BMI, operative details, length of stay, preoperative staging, operation time, pathological staging, readmissions, and complications using the Clavien-Dindo classification (Clavien et al., 2009) (Q10).

Modular training
In line with the literature, consensus was reached that educational curricula for robot-assisted gynaecological surgery should be in stages (Q11), each with theoretical and practical exams. Trainees should pass each module before commencing the next (Q17).

Basic training
Basic training should include baseline evaluation, e-learning, simulation based training, procedure observation, and team simulation (Q12).
• A baseline evaluation should help group novices by their theoretical knowledge and pre-existing skills. For this purpose, a written test and VRsimulation were recommended (Q13).
• E-learning should include information on patient selection, port placement, docking, trouble shooting, link to FRS, theatre team efficiencies, non-technical skills and standardised emergency management (Q14). It was recommended that required operating room observations should be volume based (Q15).
• Basic simulation training should include VRsimulation and dry-and wet-lab teaching (Q16).
• Team training should include emergency scenarios, team decision making, bedside assistance, docking and patient turnaround (Q21). Among the CanMed roles -which SERGS subscribes to -leadership is regarded as an important nontechnical-skill (Frank et al., 2015). No consensus was reached on recommendations for the content of non-technical skills training (Q20).

Advanced training
Analogous to basic learning, consensus was reached to perform advanced training in a modular/ stepwise approach. E-learning on index procedures was recommended supplemented by video demonstrations, access to video libraries, simulation, offered commercially to monitor the performance of individual robotic surgeons (White et al., 2015;Polin et al., 2016).

Volumetric criteria
The portfolio with subsequent assessments avoids defining a volume criterion for certification. Various studies have resulted in volume criteria that range from 10 to 100 procedures necessary to reach proficiency (Pitter et al., 2008;Brinkman et al., 2012;Letouzey et al., 2014;Sinno and Fader, 2014;Conrad et al., 2015;Nezhat and Lakhi, 2016;Brinkman et al., 2017). However, certification should not be based on numbers only but predominantly on assessment of competence (Brinkman et al., 2012(Brinkman et al., , 2017.

Results of the Delphi process
Consensus was reached in multiple areas of robotic education, qualification, course and content of training, structure of curriculum, and assessment tools ( Table 1). Among all panel members there was agreement that a standardised training curriculum for gynaecology would be advantageous for robotic assisted gynaecological surgery (Q1).

Requirements for the trainer/proctor
Consensus was reached that trainers should be accredited. There were no suggestions on the content and instruments for trainer-certification nor on its implementation (Q2).

Requirements for the Educational Training Centre
Consensus was reached that training centres should be accredited by a recognised society (Q7). Agreement was reached that accreditation of centres should be based on case volume (Q8) and expertise of the trainer (Q9). Although hard data are lacking, the panel agreed on a minimal requirement of over 100 cases/year per center (Q9) as a prerequisite for accreditation.

Requirements for qualification as an independent surgeon
Consensus was reached on a minimum of ten mentored cases before a trainee should work independently (Q4). Furthermore, experienced surgeons should continue to be tested on the advanced curriculum (Q3), although this issue was not part of the reviewed statements. To qualify for certification, a video of the index procedure in addition to a completed logbook (Q39) should be submitted to the society for review.
It was felt that experienced surgeons did not did not address criteria for certifying an examiner.
• use of validated scoring systems to analyse videos (Q36). Simultaneously, the panel could only make a general recommendation for the use of subjective and objective scoring systems (Q37).

Discussion
The Delphi method structures group communications to process complex problems (Thangaratinam, 2005;Collins et al., 2016). It is used to gain and aggregate expert opinions on issues "where hard data is unavailable" (Mahajan et al., 1976). Seven members are considered as "a suitable minimum panel size" for a Delphi-process, but sizes vary between 4 to 3000 (Mahajan et al., 1976;Thangaratinam, 2005). In the end, the panel size will be subject to the availability of dedicated experts. In the context of minimal invasive gynaecological surgery the Delphi-method has been used to define assessment of laparoscopic gynaecological procedures such as hysterectomy (Tremblay et al., 2014).
We used the results of a literature search as a backbone for formulating topics of discussion, called herein issues. Through the Delphi process, major issues in training of robotic surgeons were identified and the minimal requirements agreed. In summary, identification of training centres is volume based, next to the availability of educational tools such as e-learning, virtual learning, model training and supervised procedural teaching. The training programme should be modular, with regular assessments to monitor progress. Unlike classical surgical training, procedural learning should be stepwise allowing a focus on each step. A portfolio should help adherence to systematic training and assessment and provide the basis for certification.
The introduction of systematic and structured learning has changed surgical training. The 'see one, do one, teach one' principle has been abandoned and assessment of surgical performance is no longer a short observation by a single tutor resulting in a brief and undocumented verdict. During training, not only technical skill is important but also other competencies are recognised as valuable for medical education and these need assessment (Frank et al., 2015). Competence based assessment is now accepted, and urologists have embraced this for robotic training (Ahmed et al., 2010). Although the Delphi consensus did not result in abandoning volume based criteria for certification, competence based assessment places the emphasis on proficiency. The evolution of competence is assessed in the portfolio.
Risks to patients during an apprenticeship can be minimised by stepwise training with hands-on modular console teaching, transition to full training, and a final evaluation (Q18).
It was agreed that advanced e-learning in a modular, stepwise approach should also take into account aspects such as patient selection and preparation, port placement, non-SERGS consensus on robotic training technical skills training, trouble shooting, emergency scenario management and knowledge of additional equipment in theatre (Q19).

Structure of Curriculum
Standardisation of educational programmes seems necessary to compare outcomes. In this context, specific aspects of the structure of a robotic curriculum were reviewed.

Target Group
It was felt that consideration of prior knowledge and experience was important (Q22). Consensus was reached that there should be a common approach for basic robotic skills training with similar pathways across subspecialties (Q23).

Target Skills
Index procedures mastered over a given time should be suitable as proof of general theoretical knowledge and practical skills of a novice (Q25). A simple hysterectomy (Q26) was deemed an appropriate index procedure for benign gynaecology and a pelvic lymphadenectomy (Q27) for gynaecological oncology. A stepwise approach (modular training) of the index procedure was regarded advantageous (Q24). It seemed necessary to have a proctor present when the trainee transitioned to full procedures in his/her institution (Q28).

Test Instruments
A validated scoring system is beneficial for the trainee to provide consistent feedback (Q33). Such instruments should have been tested for face-and construct-validity. Questions may be used as instruments for testing knowledge at each educational level (Q29). For testing theoretical knowledge on different educational levels the panel agreed that • online tests are suitable instruments for evaluating progress for advanced learning modules (Q30).
• efficiency of non-technical-skills training should be assessed with a scoring system (Q31). The use of the NOTSS-System is recommended (Q32).
For assessment of procedural progress it was advised to • evaluate the full operation with a submitted video (Q34) by two certified "expert" independent examiners (Q38). No consensus was reached on case numbers to be sent for analysis (Q35). The survey the trainers (Collins et al., 2019). This needs to be developed to secure excellent and safe care of our patients.

Conclusion
In conclusion, immediate implementation of a structured curriculum is recommended. Guidance for training is needed as stricter regulation and monitoring of surgeons is demanded. There is increasing awareness that the safe introduction of new technology is the responsibility of individual institutions and care providers (Lee et al., 2011;Pradarelli et al., 2017). Guidance will assist implementing standardised and adequate educational programs.
learning in a dry and wet laboratory before embarking on a real-life procedure. E-learning modules have been developed to prepare for hands-on training (Maertens et al., 2016). Virtual training modules have been developed for technical and procedural training (Julian et al., 2018). Box training for technical instruction and development of hand-eye co-ordination has been validated (Stefanidis et al., 2011). Finally, performance during real-life surgery can be evaluated objectively using assessment tools such as OSATS (Faulkner et al., 1996).
There are several drawbacks of the process leading to the development of a SERGS' training programme. Firstly, the size of the expert panel is small because there are relatively few gynaecological surgeons regularly using a robot and being involved in training in this new technique, the number of panel member numbers is low. At the same time this reflects the urgency of such a curriculum in order to promote expertise in the robotic field. As issues addressed were also reviewed in the literature, this was not felt to cause major bias. Secondly, the literature on training in gynaecological robotassisted surgery is limited. The general principles of medical education also apply to specific training in robot-assisted surgery. Therefore, these principles were included also to obtain a representative view of surgical training.
In preparation, SERGS drafted a pilot curriculum in the form of a fellowship-programme with four robotic novices trained in four high-volume centres of excellence (Rusch et al., 2018). The curriculum was standardised with a modular and stepwise educational programme and used validated tests as proof of efficacy. This limited experience proved a need for more in depth evaluations of various educational issues, as well as the need for close monitoring of curriculum adherence. In particular, it revealed trainers were generally unacquainted with educational tools and should be trained themselves.
This Delphi process provides minimal requirements for a suitable programme. It has been the basis for the SERGS endorsed curriculum with clear outlines of training needs including assessment tools. For the index procedures, it details steps that need to be taught (Supplementary Material, Appendix 1; link: https://www.sergs.org/wpcontent/uploads/2015/08/SERGS-Curriculum-Final. pdf). This curriculum needs validation but could be used without because the Delphi process defined minor variations only in the recently validated ERUS curriculum (Volpe et al., 2015).
Finally, it should be acknowledged that this process of curriculum development has not touched on the issue of maintenance of profiency and gouvernance, neither on the issue of training of Funding: Covidien, a Medtronic Company, Neuhausen am Rheinfall, Switzerland, facilitated the Delphi process by covering meeting and travel expenses. Members of the expert panel did not receive personal fees, nor did their institutions receive vacancy fees.